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EVAL  FUNCTIONAL  DESCRIPTION 


1.0  INTRODUCTION 


1.1  Purpose  of  the  Functional  Description 

This  Functional  Description  provides  a  technical  de¬ 
lineation  of  the  specific  functions  that  EVAL  must  perform. 
It  serves  as  a  formal  basis  for  mutual  understanding  between 
the  functional  designer  of  the  system  and  the  software 
development  personnel.  Together  with  the  EVAL  System  Speci¬ 
fication,  it  serves  as  the  basic  documentation  for  systems 
development  and  implementation. 

1.2  References 


1.2.1  Barclay,  S.,  et  al.  Handbook  for  Decision 
Analysis.  Technical  Report  77-6-30.  McLean, 
Virginia:  Decisions  and  Designs,  Inc.,  Sep¬ 
tember  1977. 

1.2.2  Allardyce,  Linda  B.;  Amey,  Dorothy  M. ; 
Feuerwerger,  Phillip  H.;  Gulick,  Roy  M. 
Documentation  of  Decision-Aiding  Software: 
EVAL  Users  Manual.  McLean,  Virginia: 
Decisions  and  Designs,  Inc.,  November  1979. 

1.2.3  Allardyce,  Linda  B.;  Amey,  Dorothy  M.; 
Feuerwerger,  Phillip  H.j  Gulick,  Roy  M. 
Documentation  of  Decision-Aiding  Software: 
EVAL  System  Specification.  McLean,  Virginia: 
Decisions  and  Designs,  Inc.,  November  1979. 
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1. 3  Terms  and  Abbreviations 


1.3.1  EVAL  -  EVAL,  the  name  of  the  system,  is  an 
abbreviation  for  Evaluation,  reflecting  the  system's  major 
area  of  applicability. 

1.3.2  Terms  -  Standard  mathematical  notations  and 
decision-analytic  terminology  are  used  throughout  this 
Functional  Description.  Decision-analytic  terms  are  defined 
when  they  are  first  encountered.  Reference  1.2.1  provides 
more  detail  on  decision  analysis,  should  it  be  desired. 
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2 . 0  SYSTEM  SUMMARY 


2.1  System  Description 

EVAL  is  a  decision-analytic,  model-building  software 
system.  Its  general  purpose  is  to  aid  decision  makers  by 
providing  them  a  capability  to  construct,  store,  retrieve, 
exercise,  and  refine  evaluation  models  of  complex  decision 
problems  they  face. 

EVAL  uses  multi-attribute  utility  (MAU)  theory  as  the 
organizing  framework  for  information  processing.  MAU  assess¬ 
ment  is  the  methodological  tool  with  which  the  decision 
maker  defines  and  exercises  EVAL  models  to  evaluate  alterna¬ 
tive  systems.  The  MAU  assessment  model  is  employed  in 
decision-analytic  studies  to  quantify  and  assess  the  overall 
relative  value  of  very  complex  multi-attributed  alternatives 
that  differ  across  a  large  number  of  characteristics. 

The  MAU  model  is  hierarchical  in  nature,  starting  with 
the  overall  top-level  criterion  for  which  a  comparative 
evaluation  score  is  desired.  That  factor  is  successively 
decomposed  into  its  component  criteria  in  descending  levels 
of  the  hierarchy  such  that  each  successive  lower-level 
criterion  is  more  specific  than  those  at  the  preceding 
level.  At  the  lowest  level  of  the  hierarchy  are  the  cri¬ 
teria  for  which  specific  scores  will  be  assessed  for  the 
alternatives  by  the  user.  For  evaluating  the  alternative 
systems,  the  criteria  at  the  lowest  level  address  the  more 
easily  understood  operational  or  technical  attributes  of  the 
systems  under  evaluation.  For  evaluating  funding  programs, 
they  may  be  the  specified  attributes  of  potential  program 
performance. 
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EVAL  permits  the  user  to  build  and  exercise  MAU  assess¬ 
ment  models.  The  overall  objective  of  EVAL  is  to  ensure 
that  the  ultimate  evaluation  choice  is  coherent:  a  choice 
that  is  fully  consistent  with  the  decision  maker's  own  value 
structure  and  belief  about  the  future  performance  of  the 
alternative  systems  or  programs  under  consideration.  For  a 
complete  description  of  the  purpose  and  use  of  EVAL,  see 
EVAL  Users  Manual,  reference  1.2.2. 

2.2  Design  Objectives 

The  EVAL  system  is  designed  to  be  used  interactively  by 
end  users  who  are  relatively  unsophisticated  with  respect  to 
computer  technology.  Accordingly,  the  design  satisfies  two 
human-factors  objectives:  EVAL  is  a  menu-driven  system,  and 
it  is  generally  forgiving  of  procedural  errors  by  the  user. 

In  addition,  to  facilitate  the  production  of  the  pro¬ 
gram  specification  and  coding  necessary  to  implement  EVAL  at 
a  physical  site,  the  system  is  designed  in  a  hierarchically 
structured  and  modular  fashion.  The  logical  structure  of 
EVAL  is  contained  in  EVAL  Systems  Specification,  reference 
1.2.3. 
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3.0  DETAILED  CHARACTERISTICS 


The  fundamental  product  of  EVAL  is  a  multi- attribute 
utility  (MAU)  assessment  model.  The  EVAL  system  enables  the 
user  to  create,  store,  retrieve,  exercise,  and  refine  MAU 
assessment  models  interactively. 

All  of  the  specific  functions  that  EVAL  performs  are 
related  to  the  EVAL  MAU  assessment  model.  Therefore,  in 
order  to  establish  a  frame  of  reference  for  understanding 
the  EVAL  functions,  it  is  necessary  to  begin  with  a  detailed 
description  of  the  format,  inputs,  and  outputs  of  the  MAU 
model.  A  description  of  the  specific  functions  that  EVAL 
performs  appears  in  Section  4.0. 

3 . 1  Model  Description 

Each  MAU  model  created  by  the  user  has  a  unique  label, 
and  each  is  constructed  by  using  the  same  generic  hier¬ 
archical  format.  A  typical  model  format  is  represented 
graphically  in  Figure  3-1.  The  format  always  consists  of 
all  of  the  following  elements  which,  when  completely  speci¬ 
fied,  uniquely  define  an  EVAL  MAU  model. 

3.1.1  The  evaluation  problem  -  A  short  label  defining 
the  problem.  This  label  distinguishes  the  MAU  model  as  a 
whole  and  is  used  to  store  and  retrieve  the  model. 

3.1.2  Criteria  -  A  set  of  criteria  constituting  over¬ 
all  evaluation.  The  overall  evaluation  criterion  is  decom¬ 
posed  into  component  criteria  which  themselves  may  be 
decomposed.  Those  criteria  which  are  not  further  decomposed 
comprise  the  bottom-level  criteria.  The  bottom-level  cri¬ 
teria  are  circled  in  Figure  3-1. 
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OVERALL  EVALUATION  CRITERION 
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Figure  3-1 

THE  FORMAT  OF  A  MULTI-ATTRIBUTE  UTILITY  ASSESSMENT  MODEL 
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3.1.3  Alternatives  -  A  list  of  the  alternative  systems 
or  programs  that  the  decision  maker  must  evaluate.  Each 
alternative  is  given  an  appropriate  label. 

3.1.4  Utility  scores  *  A  list  of  the  scores  repre¬ 

senting  the  relative  utility  of  each  alternative  evaluated 
with  respect  to  each  bottom-level  criterion.  A  utility  is  a 
number  between  zero  and  one  that  represents  the  relative 
degree  of  satisfaction  of  an  alternative  with  respect  to  a 
particular  bottom-level  criterion.  Utilities  are  normally 
expressed  as  percentages  of  complete  satisfaction.  For 
example,  three  alternative  systems  might  be  scored  50  100  0 

with  respect  to  criterion  2.1.2.  The  scoring  indicates 
that,  relative  to  only  the  three  alternatives,  the  second  is 
the  best,  the  third  is  the  worst,  and  the  first  is  midway 
between  the  two  with  respect  to  that  criterion. 

3.1.5  Relative  importance  weights  -  These  weights 
describe  the  relative  importance  of  the  lower-level  criteria 
into  which  a  higher- level  criterion  is  decomposed.  A  set  of 
weights  is  assigned  for  each  decomposition;  one  weight  for 
each  criterion.  Every  criterion  except  for  the  overall 
(top-level)  criterion  must  be  assigned  an  importance  weight. 
The  weights  themselves  represent  percentages  of  the  whole 
and  must  sum  to  100%.  However,  unnormalized  weights  may  be 
specified,  in  which  case  EVAL  will  then  normalize  them  to 

sum  to  100%.  For  example,  unnormalized  weights  of  50  100  50 

would  be  normalized  to  25  50  25. 

3.1.6  Data  identification  numbers  -  These  are  assigned 
to  each  criterion  and  describe  how  all  of  the  criteria  are 
logically  related.  This  numerical  labeling  process  is  shown 
in  Figure  3-1.  For  example,  the  subfactors  of  criterion  1 
have  data  identification  numbers  1.1  and  1.2. 
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The  model  is  completely  and  uniquely  defined 
when  all  of  the  above  factors  are  specified. 

3.2  Results  of  the  Model 


The  input  specifications  describing  the  model  can  be 
processed  to  produce  the  following  results  of  interest  to 
the  user. 


3.2.1  Overall  results  -  The  overall  utility  associated 
with  each  alternative.  For  each  alternative,  the  overall 
utility  is  obtained  by  weighting  and  adding  the  utility 
scores  assigned  to  the  bottom-level  criteria  and  continuing 
to  aggregate  from  bottom  to  top.  Scores  for  any  specified 
higher-level  criterion  are  calculated  and  displayed  in  the 
same  manner. 


> 


) 


I 


3.2.2  Normalized  weights  -  A  set  of  vectors  corre¬ 
sponding  to  the  relative  importance  weights  of  the  criteria. 
A  set  of  relative  importance  weights  is  assigned  to  the 
criteria  comprising  any  specific  higher- level  factor.  These 
weights  are  normalized  into  percentages  by  dividing  each 
assigned  weight  by  the  sum  of  the  weights  of  the  component 
criteria.  That  is. 


W. 


x 


where  W  is  the  normalized  weight  of  criterion  i,  and  w  is 
the  unnormalized  weight.  In  calculating  results,  the  nor¬ 
malized  weights,  expressed  as  fractions,  rather  than  the 
relative  importance  weights,  are  used. 


) 
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3.2.3  Intermediate  results  -  A  set  of  utilities  assigned 
to  any  of  the  intermediate  criteria  during  the  calculation 

of  the  overall  result. 

3.2.4  Cumulative  weights  -  A  set  of  weights  corre¬ 
sponding  to  the  relative  importance  weights  of  the  criteria. 
These  are  calculated  as  follows:  The  top-level  criteria 
comprising  the  overall  evaluation  have  cumulative  weights 
equal  to  their  normalized  weights.  At  the  next  lower  level, 
the  criteria  are  assigned  a  cumulative  weight  computed  by 
multiplying  the  normalized  weight  by  the  cumulative  weight 
of  the  factor  to  which  it  is  attached,  and  dividing  the 
product  by  100.  This  process  is  continued  down  through  the 
structure  until  all  criteria  have  been  assigned  cumulative 
weights.  The  cumulative  weight  (CUMWT)  indicates  the  rela¬ 
tive  importance  of  the  criterion  to  the  overall  evaluation. 

3. 3  Sensitivity  Analysis 

Another  set  of  results  can  be  obtained  by  using  the 
EVAL  software.  An  n  x  11  matrix  can  be  generated  that 
displays  eleven  overall  score  vectors.  The  matrix  is  a 
sensitivity  analysis,  calculated  as  follows:  The  user 
identifies  one  criterion  of  interest  and  assigns  the  maximum 
and  minimum  cumulative  weights  that  it  may  assume.  EVAL 
varies  the  cumulative  weight  of  the  criterion  in  increments 
the  size  of  one-tenth  of  the  difference  between  the  maximum 
and  minimum  weights.  The  other  weights  in  the  model  main¬ 
tain  their  previously  assigned  proportional  relationships 
with  one  another.  That  is, 


where 


=  a  variable,  the  test  weight  of  the  criterion 
of  interest,  varied  incrementally. 


Wj  =  a  variable,  the  correct  test  weight  for  any 
other  criterion  as  is  varied. 

w.  =  a  constant,  the  originally  specified  weight 
of  criterion  i. 


w 


j 


a  constant,  the  originally  specified  weight 
of  criterion  j. 


Normally,  the  alternative  that  receives  the  highest 
overall  utility  will  change  as  the  criterion  weight  is  incre¬ 
mented  from  W  .  to  W  .  The  points  of  chancre  are  referred 
min  max 

to  as  threshold  points  and  are  easily  noted  in  the  matrix. 
Figure  3-2  shows  an  example  threshold  matrix.  Note  that  the 
alternative  having  the  highest  utility  is  identified  with  an 
asterisk. 


1.2  PERFORMANCE  CURRENT  CUMWT:  55.00 


WEIGHT 

A 

B 

C 

D 

E 

.0 

63 

54 

50 

68 

74* 

10.0 

63 

56 

52 

66 

73* 

20.0 

64 

59 

54 

64 

72* 

30.0 

64 

62 

56 

62 

71* 

40.0 

65 

64 

58 

60 

70* 

50.0 

65 

67 

60 

58 

69* 

60.0 

66 

70* 

62 

56 

68 

70.0 

66 

72* 

63 

54 

67 

80.0 

67 

75* 

65 

52 

66 

90.0 

67 

78* 

67 

49 

65 

100.0 

68 

81* 

69 

47 

64 

Figure  3-2 

SENSITIVITY  ANALYSIS 
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4.0  EVAL  FUNCTIONS 


EVAL  is  designed  to  perform  the  basic  functions  de¬ 
scribed  below.  The  functions  are  divided  into  two  separate 
categories,  STRUCTURE  and  RUN,  each  designed  to  perforin 
various  functions.  A  description  of  the  detailed  logical 
design  of  the  EVAL  functions  is  contained  in  the  manual, 
EVAL  System  Specification,  reference  1.2.3. 

4.1  Functions  Performed  Both  in  STRUCTURE  and  RUN 


4.1.1  Maintain  a  library  of  EVAL  models  -  Store  vari¬ 
ous  evaluation  models,  filed  by  their  associated  labels. 

4.1.2  Load  an  existing  EVAL  model  -  Display  the  labels 
of  those  evaluation  models  stored  in  the  model  library,  and 
permit  the  user  to  retrieve  any  desired  model.  The  loaded 
model  is  referred  to  as  the  current  model. 

4.1.3  Save  the  current  model  -  Permit  the  user  to  add 
the  current  model  to  the  model  library,  or  to  replace  an 
existing  model  with  the  current  model. 

4.2  Functions  Performed  only  in  STRUCTURE 

4.2.1  Create  a  new  EVAL  model  -  Permit  the  user  to 
create  a  new  model,  which  then  becomes  the  current  model. 

The  user  creates  a  model  by  specifying  the  elements  listed 
in  Section  3.1.  Alternatively,  the  user  may  create  separate 
sections,  or  branches  of  the  model  individually,  later 
adding  those  sections  to  the  overall  model  while  creating 
it. 


4.2.2  Revise  the  structure  of  an  existing  model  - 
Permit  the  user  to  make  changes  to  the  logical  structure  of 
the  current  model.  The  user  may: 

a.  add  new  criteria  to  the  existing  structure; 

b.  prune  (remove  entire  sections  from)  an 
existing  structure; 

c.  edit  criterion  names; 

d.  alter  the  logical  relationship  among  existing 
criteria  by  changing  data  identification 
numbers;  and 

e.  remove  any  criterion. 

4.2.3  Develop  the  structure  of  a  newly  created  or 
edited  model  -  Organize  the  criterion  data  identification 
numbers  and  labels  in  a  structural  format  to  allow  for  per¬ 
manent  storage  of  the  model,  and  for  the  input  of  utility 
scores  and  importance  weights  as  part  of  the  RUN  functions. 

4.2.4  Print  a  review  sheet  -  Print  out  the  structure 
of  the  model  thus  permitting  the  user  to  review  the  current 
model  to  determine  whether  any  structural  revision  is  re¬ 
quired.  A  sample  review  sheet  is  shown  in  Figure  4-1. 

4.3  Functions  Performed  only  in  RUN 

4.3.1  Supply  new  values  to  the  current  model  -  Permit 
the  user  to  supply  an  entire  set  of  criteria  weights  (which 
the  computer  automatically  normalizes)  or  bottom-level 
criterion  utility  scores  to  the  current  model. 
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0  OVERALL 

1  TRAINING 

1.1  INDIV  DEF 

1.1.1  TRNG  SYSTM 

1.1. 1.1  TIME 

1.1. 1.2  QUALITY 

1.1. 1.2.1  RELEVANCE 

1.1. 1.2. 2  MOTIVATION 
1.1. 1.3  AVAIL 

1.1.2  INSTRUCTOR 

1.2  MIS  DEG,T 

1.2.1  INDIV  SKL 

1.2.2  TEAM  SKL 

1.2.3  EMPLOY  SKL 

1.3  QUALIFIED 

1.3.1  QUAL  OPPTY 

1.3.2  MST  STDS 

2  PERSONNEL 

2 . 1  PERS  DEF 

2.1.1  REENL  RATE 

2.1.2  PERS  RETN 

2.2  E-5/9  PCT 

2.2.1  TRNG 

2.2.2  CAR  REENL 

2.3  MIS  DEG,P 

2.3.1  CRIT  SKILL 

2.3.2  MGT  SKILL 

2.3.3  PERF  QUAL 

3  EQUIPMENT 

3.1  EQ  DEF 

3.1.1  REP  MAINT 

3. 1.1.1  MAINT  SKL 

3. 1.1. 2  MAINT  AIDS 

3.1.2  EQ  DESIGN 

3.1.3  PREV  MAINT 

3 . 1 . 3 . 1  MAINT  SKL 

3. 1.3. 2  MAINT  AIDS 

3.2  MIS  DEG,E 

3.2.1  EQ  REDNCY 

3.2.2  EMPLOY  SKL 

3.3  PCT  OP  RDY 

3.3.1  NOR, MAINT 

3.3.2  NOR, SUPPLY 

4  MATERIEL 

4 . 1  MAT  SHORT 

4.1.1  SUP  SKL 

4.1.2  MAT  USE 

4.2  PCT  ALLOW 

4.2.1  EQUIP  USE 

4.2.2  EXT  SPT 

4.3  MIS  DEG,M 

4.3.1  EMPLOY 

4.3.2  EQUIP  DES 


Figure  4-1 

A  SAMPLE  REVIEW  SHEET 


4.3.2  Print  a  worksheet  -  Provide  a  printed  worksheet 
on  which  the  user  may  record  an  entire  set  of  weights  and 
utility  scores.  A  sample  worksheet  is  shown  in  Figure  4-2. 

4.3.3  Display  the  results  of  the  current  model  - 
Permit  the  user  to  examine  the  structure  and  content  of  the 
current  model  by  displaying: 

a.  normalized  criteria  weights; 

b.  cumulative  criteria  weights; 

c.  bottom-level  criteria  utility  scores; 

d.  overall  and  intermediate  aggregated  utility 
scores; 

e.  criteria  names;  and 

f.  hierarchical  relationships  of  the  criteria 
within  the  overall  model  structure. 

A  printed  version  of  any  of  these  results  may  also  be  obtained. 

4.3.4  Perform  sensitivity  analyses  -  Permit  the  user 
to  test  the  sensitivity  of  the  overall  result  of  the  current 
model  to  changes  in  the  importance  weights  of  individual 
criteria  at  any  level  except  the  top  factor. 

4.3.5  Revise  values  assigned  to  the  current  model  - 
Permit  the  user  to  make  changes  to  the  current  model.  The 
user  may  revise: 

a.  criteria  importance  weights;  and 

b.  bottom- level  criteria  utility  scores. 
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ENTER  DATA  IN  THE  ORDER:  NOW  FUT 


0  -  OVERALL  (WT: 

1  -  TRAINING  (WT: 

1.1  -  INDIV  DEF  Twi 

1.1.1  -  TRNG  SYSTM 

1.1. 1.1  -  TIME 

1.1. 1.2  -  QUALITY 

1.1. 1.2.1  -  RELEVANCE 

1.1. 1.2. 2  -  MOTIVATION 


) 


(WT: 


(WT: 

(WT:' 


(WT: 

(WT: 


1.1.1. 

,3  -  AVAIL 

(WT: 

1.1.2 

-  INSTRUCTOR 

(WT: 

T 

1.2  - 

MIS  DEG,T 

(WT: 

T~ 

1.2.1 

-  INDIV  SKL 

(WT: 

) 

1.2.2 

-  TEAM  SKL 

(WT:* 

“> 

1.2.3 

-  EMPLOY  SKL 

(WT:" 

I) 

1.3  - 

QUALIFIED 

(WT: 

T" 

1.3.1 

-  QUAL  OPPTY 

(WT: 

) 

1.3.2 

-  MST  STDS 

(WT:* 

'_) 

2  -  PERSONNEL  (WT 

:  ) 

2.1  - 

PERS  DEF 

(WT: 

_) 

2.1.1 

-  REENL  RATE 

(WT: 

) 

2.1.2 

-  PERS  RETN 

(WT:* 

I) 

2.2  - 

E-5/9  PCT 

(WT: 

T” 

2.2.1 

-  TRNG 

(WT: 

) 

2.2.2 

-  CAR  REENL 

(WT:* 

I) 

2.3  - 

MIS  DEG,P 

(WT: 

7“ 

2.3.1 

-  CRIT  SKILL 

(WT: 

) 

2.3.2 

-  MGT  SKILL 

(WT:* 

') 

2.3.3 

-  PERF  QUAL 

(WT:* 

“) 

3  -  EQUIPMENT 
3.1  -  EQ  DEF 


(WT: _ ) 

(WT: 


3. 

3. 

3. 

3. 

3. 

3. 

3. 

3. 

3. 

3. 

3. 

3. 

3. 

4  - 
4.1 


1. 

1. 

1. 

1. 

1. 

1. 

1. 

2 

2. 

2. 

3 

3. 

3. 


1  - 


2  - 
3  - 


4. 

4. 

4. 

4. 

4. 

4. 

4. 

4. 


1. 

1. 

2 

2. 

2. 

3 

3. 

3. 


REP  MAINT 

1.1  -  MAINT  SKL 

1.2  -  MAINT  AIDS 
EQ  DESIGN 
PREV  MAINT 

3.1  -  MAINT  SKL 

3.2  -  MAINT  AIDS 

-  MIS  DEG,E 

1  -  EQ  REDNCY 

2  -  EMPLOY  SKL 

-  PCT  OP  RDY 

1  -  NOR, MAINT 

2  -  NOR, SUPPLY 
MATERIEL 

-  MAT  SHORT 

-  SUP  SKL 

-  MAT  USE 
PCT  ALLOW 

-  EQUIP  USE 

-  EXT  SPT 
MIS  DEG,M 

-  EMPLOY 

-  EQUIP  DES 


Figure  4-2 
A  SAMPLE  WORK  SHEET 
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